Tracking Tools Teach You How To Determine If A US Server Is Offline And The Current Impact Area

1. Preparation and concept explanation

1) First, clarify the definition of "network disconnection": the server cannot respond at all (ICMP/TCP/HTTP are unreachable) or partial routes are unreachable (only inaccessible to specific regions).
2) Prepare basic tools: ping, traceroute (or tracert), mtr, curl, telnet, dig/nslookup.
3) Advanced tools: RIPE Atlas probe, ThousandEyes, BGP Looking Glass, BGPStream, Hurricane Electric Looking Glass.
4) Related concepts: BGP revocation/black hole, DNS resolution issues, CDN cache hits, ISP link failures, DDoS flooding.
5) Data collection requirements: record timestamps, source IPs, destination IPs, return codes, packet loss rates, RTTs, and AS/ASN information per hop.
6) Prepare controlled probes in advance (e.g., US multi-region VPS or RIPE Atlas probe ID) to facilitate cross-region comparative testing.

2. Basic screening steps (from local to global).

1) Locally, first use ping to test the target IP (e.g., server IP 34.216.45.123) to record packet loss rate and average RTT.
2) Use traceroute/mtr to check the routing path and locate at which jump starts packet loss or timeout.
3) Test TCP/HTTP ports (such as 80/443/22) with curl or telnet to confirm whether the service layer is alive.
4) Use dig to check if domain resolution is correct, check A/AAAA, CNAME, TTL, and whether DNS has been corrupted.
5) Request status from external monitoring points (UptimeRobot, Pingdom) to confirm whether it is a local user issue.
6) If DDoS is suspected, check indicators such as server traffic peaks, connection counts, iptables/nf_conntrack overflow, etc.

3. Use distributed probes to determine the impact area

1) Use RIPE Atlas or ThousandEyes to initiate detection in North America, South America, Europe, Asia-Pacific, and other regions, and compare success rates with RTT.
2) Use multiple cloud VPSs (us-east-1, us-west-2, us-central-1) as probes for ping/traceroute. Example probe IP: 3.231.12.45(us-east-1), 34.220.78.101(us-west-2).
3) Record the packet loss rate from each probe to the target. If only the eastern US probe loses packet while Europe is normal, it is inferred to be a regional fault.
4) Use BGP Looking Glass to check whether the target server AS (e.g., AS16509 for AWS) has large-scale revocation announcements.
5) Check whether DNS resolution matches across regions, and whether CDN nodes are still cached and responsive.
6) Comprehensive judgment: If probes in multiple regions are unreachable, it indicates a global fault; If only a few areas are inaccessible, it is due to local ISP or backbone link issues.

4. Case Analysis: Replay of Real Events (Example).

1) Timeline: 2023-09-14 09:10 UTC, monitoring alarm: us-east-1 multiple VPS HTTP 503/timeout.
2) Preliminary data: Internal monitoring shows TCP SYN is not completed, and ICMP packet loss rate has risen to 90%.
3) The traceroute shows that at the 6th hop (ASN 703 - broadband backbone), packet loss starts at 100%, and the AS path is reset.
4) BGP monitoring (BGPStream) shows peer-to-peer route refreshes during the same time period, with some prefixes withdrawn.
5) Result: After contacting the upstream ISP, it was confirmed that the core switch software failure caused several hours of regional outages. After recovery, the service gradually rebounded.
6) Lesson: Cross-region backup and multiple upstream providers are needed to avoid single points of failure; enable any cast-based CDN like Cloudflare if necessary to reduce impact.

5. Data Demonstration: Examples of cross-region probe results

The table below shows the simulated probe test results (Time: 2024-06-12 14:00 UTC), showing packet loss rates and average RTT (ms) from different regions to target IP 34.216.45.123.

probe position	source IP	packet loss rate	average RTT (ms)	Note
US-East (Virginia)	3.231.12.45	92%	120	Massive packet loss, possibly a regional backbone failure
US-West (Oregon)	34.220.78.101	10%	85	Intermittent packet loss
Europe (Frankfurt)	18.194.23.56	0%	180	stable and reachable
Asia (Tokyo)	13.112.33.77	0%	230	stable and reachable

1) The table shows that the US East Probes have severe packet loss, while Europe/Asia-Pacific are normal, indicating a regional network failure.
2) Check BGP and the upstream provider's status page to confirm whether there are any large-scale revocations or link congestion.
3) If the target is the CDN backend, edge node responses must be detected simultaneously to eliminate origin station issues.
4) Record this data for post-archiving and SLA claims.
5) Automatically collect these table data during failures, facilitating quick decision-making and communication.

6. Protection and mitigation recommendations (including server configuration examples).

1) Server configuration example: VPS configuration: 8 vCPU, 16 GB RAM, 200 GB NVMe, network bandwidth 1 Gbps, operating system Ubuntu 20.04, deployed Nginx + Keepalived for high availability.
2) Multi-region redundancy: deploy at least two regions (us-east-1, us-west-2), and use cross-zone read-only replication and periodic failover drills for databases.
3) DDoS protection: Enable Cloudflare/CloudFront or AWS Shield Advanced to clean traffic peaks and set rate-limits and black hole routing.
4) Network Monitoring and Alarms: Prometheus + Grafana collects host network metrics and combines UptimeRobot/ThousandEyes for external synthetic monitoring.
5) BGP and DNS redundancy: Multiple upstream ASs and use Anycast DNS (such as NS1, Cloudflare DNS) to reduce the risk of single points of failure.
6) Disaster recovery drills and SOPs: Write fault diagnosis SOPs (including steps for using traceroute, RIPE Atlas, and BGP viewers), and regularly practice and record them.